IBM Contributions to Computer Performance Modeling

blocks of GPSS, CSS blocks are predefined characterizations of IBM hardware components. Construction of the simulation model in CSS is thus primarily a matter of describing the software (operating system, application programs, etc.) which runs on the hardware. A further step in this direction of alleviating the need for simulation programming is the Systems Network Analysis ProgrardSimulated Host Overview Technique (SNAPISHOT) [49]. In addition to characterizations of hardware components, SNAP/SHOT provides corresponding characterizations of many of IBM’s software products, especially those associated with the Systems Network Architecture. Research Queueing Package An alternate approach to alleviating the need for simulation programming is the high-level modeling language provided by the Research Queueing Package (RESQ) [50-521. The three principal contributions of RESQ are that 1) Several solution methods, numerical, approximate, and simulation, are brought together in one software package. 2) Systems are described in terms of very high-level abstract elements, based on queueing networks. 3) Several user interfaces provide both novice and experienced users productive means to describe systems. We discuss each of these contributions in turn. In the past many (most?) modeling practitioners have restricted themselves to one solution method, either analytic (including numerical and approximate methods) or simulation. RESQ includes the previously mentioned QNET4 package, approximate solution components, and a simulation component for the solution of models. Thus the RESQ user is strongly encouraged to avoid arbitrary restrictions on solution methods and to use a method most appropriate to the problem at hand. The presence of several solution methods also makes feasible the mechanical use of the hybrid solution methods to be described. There are a number of assumptions required for exact analytic or numerical solution of a significantly sized YONATHAN BARD AND CHARLES H. SAUER IBM J. RES. DEVELOP. 0 VOL. 25 NO. S SEPTEMBER 1981 queueing network model to be feasible. In addition to specific limitations on particular types of queues, there are general assumptions usually left implicit, e . g . , that a job in the queueing network contends for only one resource at a time and/or that a job may not be involved in simultaneous synchronous activities. However, characteristics such as these are important in actual systems, where several resources (e .g . , memory, channel, controller and device) may be necessary for particular activities and messages may be transmitted as packets across different communication paths, to be reassembled at their destination. RESQ provides extensions of traditional queueing networks so that such characteristics may be included in a model to be solved by approximation or simulation. The most important of these extensions is the “passive” queue, first proposed by Foster, McGehearty, Sauer, and Waggoner [53] and redefined in RESQ [50, 51, 541. Traditional queues are referred to as “active” because a job holding a resource of the queue is actively using the resource. Resources of passive queues are held so that the job may use a resource of primary importance. Passive queues have been demonstrated to provide compact representations of complex contention situations and protocols [5 1 , 551. The first RESQ interface was an interactive prompter, with built-in tutorial facilities, so that a novice could easily learn RESQ terminology and characteristics. Recently, a second version of RESQ has been developed, compatible with the first [56]. This version incorporates a much more sophisticated user interface, a modeling language analogous to a programming language. The RESQ2 language has been designed to encourage the user to produce well-structured models, in the sense of structured programming, so that modelers can effectively cope with large systems. The “submodel” facility of RESQ2 is designed so that modelers can cooperate in constructing a model and build upon the previous work of other modelers. Workload characterization Trace-driven modeling In simulation models, the workload has traditionally been described by means of probability distributions. These, however, may not capture important interdependencies of workload characteristics. This problem is overcome in trace-driven modeling, first proposed by Cheng [57] and later popularized by others [58-601. With trace-driven modeling, the simulator “executes” the same sequence of transactions that was actually traced on a real system. If used properly, trace-driven models can reproduce measurement results very closely. One can then make modifications in the model and have confidence that the model results are very close to the [BM J. RES. DEVELOP. VOL. 25 NO. 5 SEPTEMBER 1981 performance that would be observed if corresponding modifications were made to .the actual system. Characterization of paging In both distribution-driven and trace-driven simulations of virtual memory systems, a compact and accurate representation of paging activity is desirable. The obvious representation, a complete history of page references, is usually impractical. Of particular impact has been the characterization in terms of distance strings in stack replacement algorithms [61]. Other related work includes the lifetime functions discussed by Belady and Kuehner [62] and the semi-Markov characterizations of Lewis and Shedler [63]. Other compact representations of paging behavior have been developed at IBM for inclusion in various models. These include the macro-instructions of Boksenbaum et al. [a], the page survival index [65], the paging index [MI, and the global LRU analysis of Chiu and Chow [60]. Statistical aspects of simulation Random number generators When one is characterizing systems by probability distributions, one must have generators for producing samples from the distributions. Nearly all practical generators for general distributions require a generator for the uniform distribution on the interval [0, I]. In designing such a uniform generator there are a number of pitfalls which can only be avoided by careful use of number theory to propose a generator and of rigorous statistical tests to verify that the generator has the desired properties [66]. The generator proposed by Lewis, Goodman, and Miller [67] has been shown to have very good properties. In fact, its properties are so highly regarded that this generator has been subsequently incorporated in highly regarded software for other vendors’ machines [68]. This is surprising because random number generators are usually designed for specific arithmetic characteristics, such as word size, and are generally not easily transported. IBM research has also contributed methods for simulation of processes not described by stationary distributions, notably nonhomogeneous Poisson processes [69]. Output analysis Another difficult problem in probabilistic simulations is the analysis of output. The running of the simulation is a statistical experiment; the results of the simulation program may not be accurate estimates of model performance measures. The most important recent contribution to this problem is the Regenerative Method for confidence intervals [70]. IBM has contributed a number of improvements and extensions for the Regenerative Method, including stopping rules [71], extension to response time distributions [3], and variance reduction techniques [72]. IBM authors have also applied variance reduction techniques to previous methods for confidence 567 Y0NATHANBARDANDCHARLESH.SAUER intervals [73]. The regenerative method is incorporated in RESQ, and the practical applicability of the method has been demonstrated by a number of RESQ models [51]. A recently proposed spectral method [74] for confidence intervals may prove to be more useful than previous methods, including the regenerative method. Computational expense Hybrid simulation One factor in the computational expense of simulation is the disparity in event rates in different parts of the system, causing unnecessarily long simulation of one part in order to have a long enough simulation of another part with low event rates. An important method to avoid this expense is hierarchical solution, where these different parts of the model are solved separately. A special case of hierarchical solution is hybrid simulation, where part of the model is solved numerically. Examples of hybrid simulation are found in [60] and [75]. Design of experiments and validation In addition to reducing the expense of individual simulations, one can reduce the number of simulations required to cover aparameter space by appropriate design of experiments[76]. Rather than run a simulation for each combination ofparameters, one can run simulations for a small subset ofthe combinations and estimate results for the other com-binations. This approach is also useful for validating themodel.For this purpose, identical sets of experiments arerun on the real system and on the model. The statisticaleffects of various system parameters are evaluated inboth cases, and the two sets of computed effects aretested for lack of significant differences. If the test ispassed, the model is considered to be validated. Modelvalidation can also be combined with estimation of theunknown model parameters[77]. References1. A. 0. Allen, Probability, Statistics, and Queueing Theory:With Computer Science Applications,Academic Press, Inc.,New York, 1978.2. H. Kobayashi, ModelingandAnalysis, Addison-WesleyPublishing Co., Reading, MA, 1978.3. D. L. Iglehart and G. S. Shedler, Regenerative Simulation ofResponse Times in Networks of Queues, Springer-Verlag,New York, 1980.4. C. H. Sauer and K. M. Chandy, Computer Systems Per-formance Modeling, Prentice-Hall, Inc., Englewood Cliffs,NJ, 1981.5 . Y. Bard, “Performance Criteria and Measurement foraTime Sharing System,” ZBM Syst. J . 10, 193-216 (1971).6. H. P. Friedman and G. Waldbaum, “Evaluating SystemChanges Under Uncontrolled Workloads: A Case Study,” 7. Y. Bard and M. Schatzoff, “Statistical Methods in ComputerPerformance Analysis,” Current Trends in ProgrammingMethodology, Volume ZZZ: Software Modeling and Its Zm-pact on Performance, K. M. Chandy and R. T. Yeh, Eds.,Prentice-Hall, Inc., Englewood Cliffs, NJ, 1978, pp. 1-51.ZBM Syst. J . 14, 340-352 (1975).8. J. R. Jackson, “Jobshop-like Queueing Systems,” Manage.Sci. 10, 131-142 (1963).9. J. Gordon and G. F. Newell, “Closed Queueing Systemswith Exponential Servers,” Oper. Res. 15, 254-265 (1967).10. A. L. Scherr, An Analysis of Time-shared Computer Sys-tems, MIT Press, Cambridge, MA, 1967.1 1 . J. P. Buzen, “Analysis of System Bottlenecks Using aQueueing Network Model,” Proceedings of the ACM-SZGOPS Workshop on SystemPerformanceEvaluation,Cambridge, MA, 1971, pp. 82-103.12. S. R. Arora and A. Gallo, “The Optimal Organization ofMultiprogrammed Multi-level Memory,” Proceedings of theACM-SZGOPS Workshop on System Performance Evalua-tion, Cambridge, MA, 1971, pp. 104-141.13. F.Baskett, K.M. Chandy, R. R. Muntz, and F. G.Palacios,“Open, Closed, and Mixed Networks of Queues with Differ-ent Classes of Customers,” J . ACM 22, 248-260 (1975).14. M. Reiser and H. Kobayashi, “Queuing Networks withMultiple Closed Chains: Theory and Computational Algo-rithms,” ZBM J . Res. Develop. 19, 283-294 (1975).15. H. Kobayashi and M. Reiser, “On Generalization of JobRouting Behavior in a Queueing Network Model,” ResearchReDort RC-5252, IBM Thomas J. Watson Research Center,